Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization
Published in Neurips, 2024
Recommended citation: Q. Shen, Y. Wang, Z. Yang, X. Li, H. Wang, Y. Zhang, J. Scarlett, Z. Zhu, and K. Kawaguchi, Memory-efficient gradient unrolling for large-scale bi-level optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
In this paper, we introduce Forward Gradient Unrolling with Forward Gradient, abbreviated as $(FG)^2U$, which achieves an unbiased stochastic approximation of the meta gradient for bi-level optimization.